{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 小熊飞桨练习册-08PaddleX底特律街景\n",
    "\n",
    "## 简介\n",
    "\n",
    "小熊飞桨练习册-08PaddleX底特律街景，是学习图像分割小项目，本项目开发和测试均在 Ubuntu 20.04 系统下进行。  \n",
    "项目最新代码查看主页：[小熊飞桨练习册](https://gitee.com/cnhemiya/paddle-workbook)  \n",
    "百度飞桨 AI Studio 主页：[小熊飞桨练习册-08PaddleX底特律街景](https://aistudio.baidu.com/aistudio/projectdetail/4237728)  \n",
    "Ubuntu 系统安装 CUDA 参考：[Ubuntu 百度飞桨和 CUDA 的安装](https://my.oschina.net/hemiya/blog/5509991)\n",
    "\n",
    "- 锯齿狼牙的预测结果，模型：BiSeNetV2\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/1f002c24960e4fe4acc24536141e709edbe79536d0484562b21fa8af1b0e25cb)\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/a43dee791b8748dfb0019d0f44fa4df4b2529b761cec4cf9bb39a5eb08b61861)\n",
    "\n",
    "\n",
    "## 文件说明\n",
    "\n",
    "|文件|说明|\n",
    "|--|--|\n",
    "|train.py|训练程序|\n",
    "|prune.py|裁剪程序|\n",
    "|quant.py|量化程序|\n",
    "|infer.py|预测程序|\n",
    "|onekey.sh|一键获取数据到 dataset 目录下|\n",
    "|onetasks.sh|一键训练，量化脚本|\n",
    "|get_data.sh|获取数据到 dataset 目录下|\n",
    "|check_data.sh|检查 dataset 目录下的数据是否存在|\n",
    "|mod/args.py|命令行参数解析|\n",
    "|mod/pdxconfig.py|PaddleX 配置|\n",
    "|mod/config.py|配置|\n",
    "|mod/utils.py|杂项|\n",
    "|mod/report.py|结果报表|\n",
    "|dataset|数据集目录|\n",
    "|doc|文档目录|\n",
    "|output|训练参数保存目录|\n",
    "|result|预测结果保存目录|\n",
    "\n",
    "## 环境依赖\n",
    "\n",
    "- [百度飞桨](https://www.paddlepaddle.org.cn/)\n",
    "- [PaddleX](https://gitee.com/paddlepaddle/PaddleX)\n",
    "- **AI Studio** 环境，右侧 **包管理** 手动安装 **PaddleX**\n",
    "\n",
    "## 数据集\n",
    "\n",
    "数据集来源于自己收集标注的百度飞桨公共数据集：[锯齿狼牙的底特律街景](https://aistudio.baidu.com/aistudio/datasetdetail/152045)\n",
    "\n",
    "数据集包含训练集，验证集，测试集，包含 MASK 掩膜 和 COCO 格式数据集，适用图像分割，语义分割，实例分割学习。 \n",
    "\n",
    "## 如何自己标注数据\n",
    "\n",
    "- 使用标注工具：[EISeg](https://gitee.com/paddlepaddle/PaddleSeg/tree/release/2.5/EISeg)\n",
    "- 中文界面，支持 MASK 掩膜 和 COCO 格式\n",
    "\n",
    "## 一键获取数据\n",
    "\n",
    "- 运行脚本，包含以下步骤：获取数据，生成图像路径和标签的文本文件，检查数据。\n",
    "- 详情查看 **onekey.sh**\n",
    "\n",
    "如果运行在本地计算机，下载完数据，文件放到 **dataset** 目录下，在项目目录下运行下面脚本。\n",
    "\n",
    "如果运行在百度 **AI Studio** 环境，查看 **data** 目录是否有数据，在项目目录下运行下面脚本。\n",
    "\n",
    "```bash\n",
    "bash onekey.sh\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 一键获取数据\n",
    "!cd work && bash run/onekey.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 一键训练、量化任务\n",
    "!cd work && bash run/onetasks.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 配置模块\n",
    "\n",
    "可以查看修改 **mod/pdxconfig.py** 文件，有详细的说明\n",
    "\n",
    "## 开始训练\n",
    "\n",
    "运行 **train.py** 文件，查看命令行参数加 -h\n",
    "\n",
    "- 示例\n",
    "\n",
    "```bash\n",
    "python3 run/train.py \\\n",
    "    --dataset ./dataset/detroit_streetscape \\\n",
    "    --epochs 32 \\\n",
    "    --batch_size 1 \\\n",
    "    --learning_rate 0.01 \\\n",
    "    --model BiSeNetV2 \\\n",
    "    --pretrain_weights \"CITYSCAPES\"\n",
    "```\n",
    "\n",
    "- 参数\n",
    "\n",
    "```bash\n",
    "  -h, --help            show this help message and exit\n",
    "  --cpu                 是否使用 cpu 计算，默认使用 CUDA\n",
    "  --num_workers         线程数量，默认 auto，为CPU核数的一半\n",
    "  --epochs              训练几轮，默认 4 轮\n",
    "  --batch_size          一批次数量，默认 16\n",
    "  --learning_rate       学习率，默认 0.025\n",
    "  --early_stop          是否使用提前终止训练策略。默认为 False\n",
    "  --early_stop_patience \n",
    "                        当使用提前终止训练策略时，如果验证集精度在early_stop_patience 个 epoch\n",
    "                        内连续下降或持平，则终止训练。默认为 5\n",
    "  --save_interval_epochs \n",
    "                        模型保存间隔(单位: 迭代轮数)。默认为 1\n",
    "  --log_interval_steps \n",
    "                        训练日志输出间隔（单位：迭代次数）。默认为 10\n",
    "  --resume_checkpoint   恢复训练时指定上次训练保存的模型路径, 默认不会恢复训练\n",
    "  --save_dir            模型保存路径。默认为 ./output/\n",
    "  --dataset             数据集目录，默认 ./dataset/\n",
    "  --train_list          训练集列表，默认 '--dataset' 参数目录下的 train_list.txt\n",
    "  --eval_list           评估集列表，默认 '--dataset' 参数目录下的 val_list.txt\n",
    "  --label_list          分类标签列表，默认 '--dataset' 参数目录下的 labels.txt\n",
    "  --lr_decay_power      默认优化器学习率衰减指数。默认为 0.9\n",
    "  --use_mixed_loss      是否使用混合损失函数。如果为True，混合使用CrossEntropyLoss和LovaszSoftmaxLoss，权重分别为0\n",
    "                        .8和0.2。如果为False，则仅使用CrossEntropyLoss。也可以以列表的形式自定义混合损失函数，列表的每一个元素\n",
    "                        为(损失函数类型，权重)元组，损失函数类型取值范围为['CrossEntropyLoss', 'DiceLoss',\n",
    "                        'LovaszSoftmaxLoss']。默认为False。\n",
    "  --align_corners       是网络中对特征图进行插值时是否将四个角落像素的中心对齐。若特征图尺寸为偶数，建议设为True。若特征图尺寸为奇数，建议设为Fal\n",
    "                        se。默认为False。\n",
    "  --backbone            图像分割模型 DeepLabV3P 的 backbone 网络，取值范围为['ResNet50_vd',\n",
    "                        'ResNet101_vd']，默认为'ResNet50_vd'。\n",
    "  --hrnet_width         图像分割模型 HRNet 的 width 网络，高分辨率分支中特征层的通道数量。默认为48。可选择取值为[18, 48]。\n",
    "  --pretrain_weights    若指定为'.pdparams'文件时，则从文件加载模型权重；若为字符串'CITYSCAPES'，则自动下载在CITYSCAPES\n",
    "                        图片数据上预训练的模型权重；若为字符串'PascalVOC'，则自动下载在PascalVOC图片数据上预训练的模型权重；若为字符\n",
    "                        串'IMAGENET'，则自动下载在ImageNet图片数据上预训练的模型权重；若为None，则不使用预训练模型。默认为'CIT\n",
    "                        YSCAPES'。\n",
    "  --model               PaddleX 模型名称\n",
    "  --model_list          输出 PaddleX 模型名称，默认不输出，选择后只输出信息，不会开启训练\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# PaddleX 图像分割模型\n",
    "# ['DeepLabV3P', 'BiSeNetV2', 'UNet', 'HRNet', 'FastSCNN']\n",
    "# DeepLabV3P backbone 网络\n",
    "# ['ResNet50_vd', 'ResNet101_vd']\n",
    "# --backbone ResNet50_vd\n",
    "# --hrnet_width 默认为48。可选择取值为[18, 48]\n",
    "# --lr_decay_epochs \"32\" --lr_decay_gamma 1 \\\n",
    "# --save_interval_epochs 4 \\\n",
    "# --pretrain_weights ./output/best_model/model.pdparams \\\n",
    "# 训练 --num_workers 2 \"IMAGENET\" \"PascalVOC\" \"COCO\" \"CITYSCAPES\"\n",
    "!cd work && python3 run/train.py \\\n",
    "    --dataset ./dataset/detroit_streetscape \\\n",
    "    --epochs 32 \\\n",
    "    --batch_size 1 \\\n",
    "    --learning_rate 0.01 \\\n",
    "    --model BiSeNetV2 \\\n",
    "    --pretrain_weights \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 查看支持的模型\n",
    "\n",
    "- 运行命令\n",
    "\n",
    "```bash\n",
    "python3 run/train.py --model_list\n",
    "```\n",
    "\n",
    "- 结果\n",
    "\n",
    "```bash\n",
    "PaddleX 图像分割模型\n",
    "['DeepLabV3P', 'BiSeNetV2', 'UNet', 'HRNet', 'FastSCNN']\n",
    "DeepLabV3P backbone 网络\n",
    "['ResNet50_vd', 'ResNet101_vd']\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 查看支持的模型\n",
    "!cd work && python3 run/train.py --model_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型裁剪\n",
    "\n",
    "- 运行 **prune.py** 文件，查看命令行参数加 -h。\n",
    "- 注意：有的模型不支持裁剪。\n",
    "- 裁剪后的精度大部分会降低。\n",
    "- 参考文档：[模型裁剪](https://gitee.com/paddlepaddle/PaddleX/tree/develop/tutorials/slim/prune)\n",
    "- 示例\n",
    "\n",
    "```bash\n",
    "python3 run/prune.py \\\n",
    "    --dataset ./dataset/detroit_streetscape \\\n",
    "    --epochs 16 \\\n",
    "    --batch_size 1 \\\n",
    "    --learning_rate 0.001 \\\n",
    "    --model_dir ./output/best_model \\\n",
    "    --save_dir ./output/prune \\\n",
    "    --pruned_flops 0.2\n",
    "```\n",
    "\n",
    "- 参数\n",
    "\n",
    "```bash\n",
    "  -h, --help            show this help message and exit\n",
    "  --cpu                 是否使用 cpu 计算，默认使用 CUDA\n",
    "  --num_workers         线程数量，默认 auto，为CPU核数的一半\n",
    "  --epochs              训练几轮，默认 4 轮\n",
    "  --batch_size          一批次数量，默认 16\n",
    "  --learning_rate       学习率，默认 0.025\n",
    "  --early_stop          是否使用提前终止训练策略。默认为 False\n",
    "  --early_stop_patience \n",
    "                        当使用提前终止训练策略时，如果验证集精度在early_stop_patience 个 epoch 内连续下降或持平，则终止训练。默认为 5\n",
    "  --save_interval_epochs \n",
    "                        模型保存间隔(单位: 迭代轮数)。默认为 1\n",
    "  --log_interval_steps \n",
    "                        训练日志输出间隔（单位：迭代次数）。默认为 10\n",
    "  --resume_checkpoint   恢复训练时指定上次训练保存的模型路径, 默认不会恢复训练\n",
    "  --save_dir            模型保存路径。默认为 ./output/\n",
    "  --dataset             数据集目录，默认 ./dataset/\n",
    "  --train_list          训练集列表，默认 '--dataset' 参数目录下的 train_list.txt\n",
    "  --eval_list           评估集列表，默认 '--dataset' 参数目录下的 val_list.txt\n",
    "  --label_list          分类标签列表，默认 '--dataset' 参数目录下的 labels.txt\n",
    "  --lr_decay_power      默认优化器学习率衰减指数。默认为 0.9\n",
    "  --model_dir           模型读取路径。默认为 ./output/best_model\n",
    "  --skip_analyze        是否跳过分析模型各层参数在不同的裁剪比例下的敏感度，默认不跳过\n",
    "  --pruned_flops        根据选择的 FLOPS 减小比例对模型进行裁剪。默认为 0.2\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# --save_interval_epochs 4 \\\n",
    "# 裁剪 --model_dir ./output/best_model --save_dir ./output/prune\n",
    "!cd work && python3 run/prune.py \\\n",
    "    --dataset ./dataset/detroit_streetscape \\\n",
    "    --epochs 16 \\\n",
    "    --batch_size 1 \\\n",
    "    --learning_rate 0.001 \\\n",
    "    --model_dir ./output/best_model \\\n",
    "    --save_dir ./output/prune \\\n",
    "    --pruned_flops 0.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型量化\n",
    "\n",
    "- 运行 **quant.py** 文件，查看命令行参数加 -h\n",
    "- model_dir 是正常训练后的模型保存目录。\n",
    "- 参考文档：[模型量化](https://gitee.com/paddlepaddle/PaddleX/tree/develop/tutorials/slim/quantize)\n",
    "- 示例\n",
    "\n",
    "```bash\n",
    "python3 run/quant.py \\\n",
    "    --dataset ./dataset/detroit_streetscape \\\n",
    "    --epochs 16 \\\n",
    "    --batch_size 1 \\\n",
    "    --learning_rate 0.001 \\\n",
    "    --model_dir ./output/best_model \\\n",
    "    --save_dir ./output/quant\n",
    "```\n",
    "\n",
    "- 参数\n",
    "\n",
    "```bash\n",
    "  -h, --help            show this help message and exit\n",
    "  --cpu                 是否使用 cpu 计算，默认使用 CUDA\n",
    "  --num_workers         线程数量，默认 auto，为CPU核数的一半\n",
    "  --epochs              训练几轮，默认 4 轮\n",
    "  --batch_size          一批次数量，默认 16\n",
    "  --learning_rate       学习率，默认 0.025\n",
    "  --early_stop          是否使用提前终止训练策略。默认为 False\n",
    "  --early_stop_patience \n",
    "                        当使用提前终止训练策略时，如果验证集精度在early_stop_patience 个 epoch 内连续下降或持平，则终止训练。默认为 5\n",
    "  --save_interval_epochs \n",
    "                        模型保存间隔(单位: 迭代轮数)。默认为 1\n",
    "  --log_interval_steps \n",
    "                        训练日志输出间隔（单位：迭代次数）。默认为 10\n",
    "  --resume_checkpoint   恢复训练时指定上次训练保存的模型路径, 默认不会恢复训练\n",
    "  --save_dir            模型保存路径。默认为 ./output/\n",
    "  --dataset             数据集目录，默认 ./dataset/\n",
    "  --train_list          训练集列表，默认 '--dataset' 参数目录下的 train_list.txt\n",
    "  --eval_list           评估集列表，默认 '--dataset' 参数目录下的 val_list.txt\n",
    "  --label_list          分类标签列表，默认 '--dataset' 参数目录下的 labels.txt\n",
    "  --lr_decay_power      默认优化器学习率衰减指数。默认为 0.9\n",
    "  --model_dir           模型读取路径。默认为 ./output/best_model\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# --save_interval_epochs 4 \\\n",
    "# 在线量化 --model_dir ./output/best_model --save_dir ./output/quant\n",
    "!cd work && python3 run/quant.py \\\n",
    "    --dataset ./dataset/detroit_streetscape \\\n",
    "    --epochs 16 \\\n",
    "    --batch_size 1 \\\n",
    "    --learning_rate 0.001 \\\n",
    "    --model_dir ./output/best_model \\\n",
    "    --save_dir ./output/quant"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 预测模型\n",
    "\n",
    "- 运行 **infer.py** 文件，查看命令行参数加 -h\n",
    "- 示例\n",
    "\n",
    "```bash\n",
    "python3 run/infer.py --model_dir ./output/best_model \\\n",
    "    --predict_image ./dataset/detroit_streetscape/0183.jpg\n",
    "```\n",
    "\n",
    "- 参数\n",
    "\n",
    "```bash\n",
    "  -h, --help            show this help message and exit\n",
    "  --model_dir           读取模型的目录，默认 './output/best_model'\n",
    "  --predict_image       预测的图像文件\n",
    "  --predict_image_dir   预测的图像目录\n",
    "  --weight              mask可视化结果与原图权重因子，weight表示原图的权重，默认 0.6\n",
    "  --result_dir          预测结果可视化的保存目录，默认 './result'\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 预测\n",
    "!cd work && python3 run/infer.py --model_dir ./output/best_model \\\n",
    "    --predict_image ./dataset/detroit_streetscape/JPEGImages/0558.jpg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 查看预测结果图像\n",
    "from PIL import Image\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "image_path = \"./work/result/visualize_0558.jpg\"\n",
    "img = Image.open(image_path)\n",
    "plt.imshow(img)\n",
    "plt.show()\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 关于图像宽高\n",
    "\n",
    "- 由于原始图像是 1920x1080 在训练前填充为 1920 的正方形图像，然后调整为 1024x1024，参见 **train.py**\n",
    "\n",
    "```python\n",
    "T.Padding(target_size=1920, pad_mode=0, im_padding_value=[0, 0, 0]),\n",
    "T.Resize(target_size=1024),\n",
    "```\n",
    "\n",
    "## 关于预测可视化的结果颜色\n",
    "\n",
    "- EISeg 颜色通道顺序为 RGB，paddlex.seg.visualize 颜色通道顺序为 BGR\n",
    "\n",
    "## 部署模型导出\n",
    "\n",
    "- 图像分割的 --fixed_input_shape 参数无效\n",
    "- 参考文档：[部署模型导出](https://gitee.com/paddlepaddle/PaddleX/blob/develop/docs/apis/export_model.md)\n",
    "- 示例\n",
    "\n",
    "```bash\n",
    "paddlex --export_inference --model_dir=./output/best_model/ --save_dir=./output/inference_model\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 部署模型导出\n",
    "!cd work && paddlex --export_inference --model_dir=./output/best_model/ --save_dir=./output/inference_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## VisualDL 可视化分析工具\n",
    "\n",
    "- 安装和使用说明参考：[VisualDL](https://gitee.com/paddlepaddle/VisualDL)\n",
    "- 如果是 **AI Studio** 环境训练的把 **output/vdl_log** 目录下载下来，解压缩后放到本地项目目录下 **output/vdl_log** 目录\n",
    "- 在项目目录下运行下面命令\n",
    "- 然后根据提示的网址，打开浏览器访问提示的网址即可\n",
    "\n",
    "```bash\n",
    "visualdl --logdir ./output/vdl_log\n",
    "```\n",
    "\n",
    "## 项目总结\n",
    "\n",
    "- 由于数据集标注的精度不好，造成模型预测的精度也比较差。\n",
    "\n",
    "## 后期改进\n",
    "\n",
    "- 重新标注，提高标注的精度，这个由于工作量比较大，完成日期遥遥无期。\n",
    "- 利用 opencv 的一些算法半自动辅助标记，这个只有一个思路。\n",
    "\n",
    "## 半自动标记思路\n",
    "\n",
    "由于不少需要标注的目标外观是比较相近的，可以先用 opencv 轮廓特征转化图像，再用这个轮廓特征去训练。前期数据标注的少可以加大训练轮数，训练集也是验证集。\n",
    "\n",
    "每批次一个外观特征相近的同一分类进行训练模型，然后用这个模型去预测其它的图片，人工验证，正确了就加入标注，这个可以写个程序辅助加入。\n",
    "\n",
    "重复以上步骤，标注图像。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 清除数据集\n",
    "!rm -r work/dataset/*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 清除模型参数\n",
    "!rm -r work/output/*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 压缩模型\n",
    "!mkdir work/bak\n",
    "!cd work && tar -caf ./bak/BiSeNetV2_32e_0.01.tar.gz ./output/best_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 解压代码\n",
    "!unzip -oq run.zip -d ./work/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
